I was looking for a solution for ages. I found this page to be helpful.

https://askubuntu.com/questions/344863/ssh-new-connection-begins-to-hang-not-reject-or-terminate-after-a-day-or-so-on

By switching to netcat it now works without a problem.